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A TUNED SINGLE PARAMETER FOR REPRESENTING 

CONJUNCTION RISK 

D. Plakalovic,* M.D. Hejduk,* R.C. Frigm*, L.K. Newman 5 

Satellite conjunction assessment risk analysis is a subjective enterprise that can 
benefit from quantitative aids and, to this end, NASA/GSFC has developed a 
fuzzy logic construct — called the F-value — to attempt to provide a statement of 
conjunction risk that amalgamates multiple indices and yields a more stable in- 
tra-event assessment. This construct has now sustained an extended tuning pro- 
cedure against heuristic analyst assessment of event risk. The tuning effort has 
resulted in modifications to the calculation procedure and the adjustment of tun- 
ing coefficients, producing a construct with both more predictive force and a 
better statement of its error. 

INTRODUCTION 

T HE identification of close approaches between two satellites in space, or conjunction assess- 
ment, is a straightforward process by which the estimated states of two closely orbiting ob- 
jects are propagated over time and compared. Although there are numerous complexities to this 
process, such as state and state uncertainty estimate techniques, propagation methods, force mod- 
eling parameters, and physical property modeling of the objects themselves, the process is objec- 
tively governed by the underlying physics and dynamics. Conversely, conjunction assessment 
risk analysis (CARA) is the interpretive evaluation of the predictions resulting from that process. 
It comprises the evaluation of satellite conjunction events against a set of risk criteria (including 
the ability to detect and predict conjunctions both confidently and accurately), the probability of 
their occurrence, and the severity should they in fact occur. Hence, effort has been devoted to this 
aspect of the satellite conjunction problem over the last decade. 

The first major milestone in the satellite conjunction risk assessment field was introducing the 
probability of collision (P c ) calculation. 12 The P c is a metric of the likelihood of the predicted 
close approach between two orbiting objects resulting in a collision between the objects’ hard- 
bodies. It considers the size of the objects involved and the geometry and predicted proximity of 
the close approach, as well as the uncertainty in the position of the objects as described by their 
estimated 3x3 position covariances. Given precise and reliable inputs to this calculation, the P c 
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should be the true assessment of conjunction risk. However, these inputs are never precisely 
known and come with their own, usually poorly characterized, uncertainties. If these inputs are 
inaccurate or unrepresentative, the result is an unrealistic assessment of the risk level and a poten- 
tial unnecessary and risky conjunction avoidance maneuver. The state estimate covariance of the 
secondary object is all too often an example of this inaccurate input. The orbit determination 
(OD) process can produce unrealistic covariances due to uncorrected biases in the input observa- 
tions, correlation among observations, and shortcomings in the force models. If one is convinced 
that unrepresentativeness of the data that feeds the P c calculation is a significant problem, then 
use of additional risk assessment parameters is necessary in order to more properly evaluate the 
risk level of a satellite conjunction event. 

In addition to the analysis of satellite conjunction risk, communicating that risk to the satellite 
Owner/Operator (O/O), who may have the ability to perform mitigative action against those risks, 
is of perhaps equal importance. As previously mentioned, the P c can be a statement of the con- 
junction risk, but its value typically fluctuates, often substantially, over the course of a conjunc- 
tion event, which makes it difficult to use as a parameter by which to communicate risk to an 
0/0, especially if the 0/0 is not familiar with the subtleties of conjunction assessment. Opera- 
tional experience at NASA Goddard Space Flight Center (GSFC) has shown that NASA O/Os 
desire a less ambiguous and more quantitative statement of the conjunction risk analysis, whether 
it be for reporting and historical purposes or for determining precautionary actions. 

In an attempt to find a comprehensive conjunction risk analysis metric, Frigm (2009) proposed 
a fuzzy logic construct - called the F-value - with the goal of capturing and conveying the con- 
junction risk analysis in a single numerical value. 3 The F-value is a rather flexible construct that 
can be used with a diverse set of O/Os to communicate conjunction risk. However, thus far this 
construct has essentially been based on heuristics, without a rigorous tuning and verification ap- 
proach. The analysis presented in this paper is an attempt to tune the F-value using empirical data, 
as well as to more precisely determine its predictive and communicative power. 



The full details of the development of the F-value construct are given in Frigm (2009), but a 
summary is provided here. Each contribution to the overall F-value is made by means of a mem- 
bership function, which is a relationship that maps the value of a certain parameter into a pre- 
established, standardized numerical scale, typically a unitless range of 0-1 or 0-10. Each fy in Eq- 
uation ( 1 ) above represents one of these mappings for a contributing parameter, such as probabili- 
ty of collision or tracking density, whereas each a, and b, represents the associated weighting 
coefficient. This approach allows parameters of different base units to be combined in a single 
expression. In the construct shown in Equation (1), the first set of terms on the right-hand side 
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reflects the conjunction’s essence by including two parameters: the smallest miss distance (among 
all three R1C* miss components) and the P C; each mapped into a 0 to 10 value, with a higher value 
ue indicating greater severity. The second set of terms on the right-hand side derive from an at- 
tempt to characterize the quality of the OD feeding the assessment by including three parameters: 
the base ten logarithm of the determinant of the combined 3x3 covariance, a quantification of 
tracking density, and the time interval between the last sensor metric observation of the secondary 
object and the creation time of the orbital conjunction message (OCM); each of these parameters 
is mapped via a membership function to a value from 0 to 1 . 

The total F-value construct is thus the product of a weighted average of the conjunction severi- 
ty terms (first term) and the OD quality terms (second term). The quality terms (bounded between 
0 and 1 ) can essentially be seen as a scaling factor applied to the actual severity terms (bounded 
between 0 and 10). The resulting F-value thus takes on a value between 0 and 10, with a higher 
value indicating higher risk. 

The initial deployment of the F-value established heuristic membership functions and set the 
weighting coefficients to unity, exploring the power of the construct with these provisional set- 
tings. The present effort seeks a more rigorous construct by reviewing and modifying the mem- 
bership function relationships and determining values of the weighting coefficients in order to 
maximize the construct’s predictive force. 

LEVELS OF CONJUNCTION EVENT SEVERITY: THE WORK TIER 

In order to tune the F-value effectively, its output needs to be compared to some external 
measure of event severity. This measure is a subjective assessment by the operational analysts 
that consistently perform this function on a daily basis. It was found that a reasonable measure of 
event severity is the amount of “work” that each event generates. Different levels of severity ne- 
cessitate different types and amounts of analytical products to be generated, so an examination of 
which particular products were produced for any given event is a good indication of the event’s 
seriousness and therefore operational risk. To allow workload rankings to be assigned, five work- 
product-based groups, called work tiers, were developed. Table 1 provides a description of each 
of these work tier levels. Work tier 4 (execution of collision avoidance maneuver) is not an ana- 
lytical product per se, but because it represents the strongest possible defensive action that can be 
taken, it seemed appropriate to make it the capstone of a set of levels intended to reflect event 
severity. Given the retained event history in the CARA operational database, it is straightforward 
to assign a given work tier level to any given conjunction event. The work tier level can thus be 
used as an indication of event severity/risk that the F-value (or any similar risk assessment con- 
struct) can attempt to predict. 


Radial, In-track, Cross-track 
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Table 1: Division of Work Tier Levels 


Work Tier Level 

Description 

0 

The CARA* team provides the Owner/Operator with a Summary Report via auto- 
mated e-mail listing all satellite conjunction events within a specified screening 
volume, including trend plots. 

1 

The CARA team notifies Owner/Operator via e-mail or phone providing the CARA 
team's assessment of the satellite conjunction event’s risk. The CARA team analyz- 
es four main figures of merit: conjunction geometry, uncertainty analysis, conjunc- 
tion evolution and trends, and orbit determination quality. 

2 

The CARA team provides a formal briefing to the Owner/Operator. The presenta- 
tion provides additional information to describe the significant attributes of the sa- 
tellite conjunction event leading to the assessment and associated risk. These in- 
clude output from the CARA team's in-house tools: trend plots, visualizations, sen- 
sitivity analyses, conjunction geometry, maneuver options, etc. 

3 

The CARA team provides a recommendation from which the Owner/Operator de- 
cides to begin the maneuver planning process. The CARA team provides assistance 
in determining possible maneuver options that would mitigate any risk associated 
with the conjunction. The CARA team also analyzes/assesses predicted post- 
maneuver conjunction events and their associated risks. 

4 

The Owner/Operator weighs the conjunction risk (from the assessment and recom- 
mendation of the CARA team) and all other mission risks and decides to perform a 
Risk Mitigation Maneuver (RMM). A debrief is typically provided. 


Experiences of the CARA team show that most O/Os prefer to receive a notification when a 
satellite conjunction event is within a certain tolerance threshold, regardless of whether the event 
is expected to become significant. This threshold, called a screening volume, is typically defined 
as a rectangular box centered on the primary spacecraft, and any secondary space object violating 
the parameters of this volume triggers a notification of a potential satellite conjunction event. Re- 
gardless of the risk severity for a particular satellite conjunction event, the notification of a 
screening volume violation is sent to the O/Os via automated e-mail. At this point, the work tier 
level of zero is assigned to the event. If the conjunction event is eventually elevated to a higher 
work tier level, it is only then treated as a significant event (work tiers 1-4); all work tier level 0 
events are considered insignificant. 

F-VALUE TUNING DATASET 

The F-value tuning process utilized 4627 unique satellite conjunction events taking place from 
July 20 1 0 to May 2011. There were 151 events with work tier level 1 or higher, while the remain- 
ing events were assigned a work tier level of 0. Each satellite conjunction event contained any- 
where from one to fifteen Orbital Conjunction Messages (OCM). An OCM is a data product that 
is delivered by the Joint Space Operations Center (JSpOC) providing useful information about a 
satellite conjunction event, including the time of closest approach (TCA), the R1C miss distances, 
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the covariance matrices, and other amplifying information. Typically, the first notification of a 
satellite conjunction event is provided approximately seven days in advance of the actual TCA. 
Subsequent OCM updates are normally delivered at a once to twice per day frequency depending 
in part on the availability of the new tracking data. Accompanying information with each OCM 
allows the calculation of the F-value. Apart from the computation of the F-value, each event’s 
work tier level assignment is also known. All of these data are preserved in the CARA operation- 
al database and were thus easily mined for the present analysis. 

F-VALUE TUNING PART I: MEMBERS HIP FUNCTIONS 

The F-value construct as presently deployed comprises four risk and three quality assessment 
parameters for which membership functions are needed, although the number of the actual risk 
parameters used is reduced to two from four through a supremum test. These seven parameters 
are: radial, in-track, and cross-track separations at the TCA; probability of collision; tracking den- 
sity (tracks used per day); time since last observation (days); and the “size” of the 3x3 combined 
covariance. The first four constitute the category of risk assessment parameters, while the last 
three are quality assessment parameters. The supremum test chooses the separation parameter (of 
the three R1C components) whose membership function evaluation indicates the highest risk. 

For several years, routine conjunction risk assessment has been performed for NASA missions 
in a variety of orbit regimes, including Low Earth Orbits (LEO), Geosynchronous Orbits (GEO), 
and Highly Elliptical Orbits (HEO). When it comes to physical RiC separations, the risk level in 
each of these orbit types is assessed differently - certain separation thresholds may be alarming in 
GEO while quite acceptable in LEO. In order to avoid defining a separate set of membership 
functions for each orbit regime, these separation parameters were normalized as a percentage of 
the maximum separation value for the given orbit regime. The maximum RIC separations com- 
monly used for screenings are 0.5 km, 5 km, and 5 km, respectively. 

The preliminary forms of membership functions were already presented in Frigm (2009). 3 
This initial concept was based on techniques utilizing the intuition and direct operational expe- 
rience of the author. Since that time, formal consultations with the entire CARA team have en- 
hanced and evolved the proposed membership functions. In addition to establishing the form, 
careful consideration of the membership function end-points was necessary, as these points estab- 
lish thresholds for which the risk essentially cannot be increased (even though in some cases the 
associated parameter value could be) and at which the parameter essentially poses no risk at all. 
The present membership function definitions are given below in Figures 1-7. 
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Figure 1: Radial Separation Membership 
Function 



Figure 2: In-track Separation Membership 
Function 




Percentage of Max Crosstrack Separation 


log (Pc) 


Figure 3: Cross-track Separation Membership 
Function 


Figure 4: Probability of Collision 
Membership Function 


The interpretation of Figures 1-7 and their use are straightforward. An OCM for a particular 
satellite conjunction event will contain all the necessary information to calculate the domain val- 
ue for each contributing parameter. For example in Figures 1-3, R1C components of the miss dis- 
tance are directly provided on the OCM. The P c value is also now being provided on the OCM, 
but that particular value was not utilized in this study. An in-house developed P c calculation algo- 
rithm was implemented. Given the dynamic range of the P c> the use of a logarithmic scale (base 
10) in this scenario is appropriate. 
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Figure 5: Tracking Density Membership 
Function (Tracks Used/Day) 



Figure 6:Time Since Last Observation 
Membership Function (Days) 


In Figures 5-6, the tracking density and the time since last observation are computed from am- 
plifying information provided within the OCM. In particular, tracking density is defined to be the 
number of tracks used per differential correction span. The time of the last observation given in 
days is a difference between the creation time of that OCM and the time of the last metric obser- 
vation of the secondary object. 



Figure 7: Combined Covariance Membership Function 

The membership functions for four of the F-value’s input parameters deploy a logarithmic 
(base 1 0) transformation of the data, which allows for the overall simpler representation as a li- 
near relationship. It bears mention that combining of the primary and secondary object’s 3x3 co- 
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variances in Figure 7 is performed in a common inertial reference frame (J2000). The determinant 
of the combined 3x3 covariance is useful because it gives a single-value representation of the 
“size” of the combined uncertainty ellipsoid. A determinant of a matrix is a product of its eigen- 
values, which for a nearly-diagonal covariance matrix is essentially the product of the uncertainty 
variances. 

The membership functions allow the calculation of all of the “raw” f-values (/• and f) in Equa- 
tion (1) for any given event. When tuning coefficients are established, an overall F-value can then 
be calculated; and the F-value, in concert with an established F-value threshold, can be used to 
determine whether an event is significant or not (work tier level greater than 0) or, potentially, the 
expected final work tier level. 

F-VALUE TUNING PART II: CANDIDATE TUNING COEFFICIENTS 

A fully defined F-value construct will have a single set of weighting coefficients, one for each 
membership parameter, and four different F-value thresholds, one for each significant work tier 
level. The calculated F-value for an event would indicate that the event belongs to a certain work 
tier if the F-value exceeds the designated threshold for that work tier. Obviously, an event with 
the F-value not exceeding the threshold of the work tier level 1 is labeled as an insignificant 
event. It is noted here that for this analysis, a representative F-value for an event is the largest F- 
value that is produced throughout the evolution of that event. 

From this formulation, two types of errors are possible when the F-value prediction fails to 
correctly determine the work tier level of a satellite conjunction event. Table 2 provides their re- 
spective definitions. 

Table 2: Type 1 and Type 2 Error Definitions 


Error 

Description 

Type 1 

This type of error is flagging a satellite conjunction event as a higher tier level ( i.e ., 
tripping the threshold for the work tier level under investigation) when in fact it is a 
lower tier level. While this error type is undesirable, its occurrence is generally pre- 
ferable to a type-2 error. 

Type 2 

This type of error is flagging a satellite conjunction event as a lower tier level (i.e., 
failing to trip the threshold for the work tier level under investigation) when in fact 
it is a higher tier level. 


It is evident that the cost function to be minimized for the optimization problem of finding the 
optimal set of F-value coefficients is a binary one, in that for any given event the F-value con- 
struct either predicts a certain work tier correctly or it does not. Such cost functions often fare 
poorly with classical constrained optimization approaches in that the process usually does not 
produce a global minimum. In other words, a small change in initial conditions results in the con- 
vergence of an iterative method to a different local minimum. Also, such minima may be identic- 
al yet be reached with substantially different coefficient sets. As an alternative method, the num- 
ber of tuning coefficient combinations is small enough that a brute force approach is possible. All 
of the tuning coefficients are systematically varied in a nested manner from 0 to 1 in the incre- 
ments of 0.1, and the candidate F-value thresholds are systematically varied from 0 to 10 also in 
the increments of 0.1. A single iteration of the brute force algorithm selects a specific combina- 
tion of tuning coefficients and an F-value threshold, and makes a binary prediction for a particular 
work tier, for all 4627 conjunction events, after which the overall count of type- 1 and type-2 er- 




rors for that coefficient and threshold set is determined. This process is repeated until all desig- 
nated coefficient combinations and F-value thresholds have been exhausted. An example of the 
results for a particular run through the brute force method for tier 1 is presented below in Figure 
8 . 



Figure 8: Work Tier 1 Prediction Region 

Ideally, if the F-value’ s predictive capabilities were flawless, then the count for both types of 
error would be zero for the set of events analyzed. Flowever, in virtually all real applications, 
there is a trade-off between the two error types, meaning that one can improve the outcome for 
one error type at the cost of worse performance for the other error type. 

It is evident that the lower frontier of the shaded region in Figure 8 gives the preferable trade- 
off between the two error types. For example, point (1100, 20), which is on the lower edge, is 
preferable to (2700, 20), because if one is to endure 20 type-2 errors, it is better that they come 
accompanied with only 1100 type-1 errors rather than 2700 type-1 errors. A curve that provides 
only the desirable edge of a trade-space such as that in Figure 8 is called a sufficient frontier. 

A sufficient frontier plot is a useful means by which one can choose the desired trade-off ratio 
between the error types. However, it must be emphasized that there is no “correct” type-1 / type-2 
error ratio, except in those rare instances in which a 0/0 ratio happens to be possible. The question 
then is of the number of additional type-1 errors that a user is willing to endure in order to reduce 
the type-2 error by a certain amount. 

Figure 9 below presents the full outcome of the brute force iterative method and gives four 
such sufficient frontier curves for each respective tier. It is helpful to remember that such curves 
do not report the results for all 4627 events because they give only the sufficient frontier; each 
eliminated point is not represented because some other point, which is shown, is necessarily a 
superior choice. 
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Figure 9: Work Tier Predictions 

The optimal set of tuning coefficients chosen is the one that results in a favorable placement 
along the sufficient frontier curves. The choice of an optimum point is of course subjective and 
depends on one’s risk tolerance and the degree of encumbrance that type-1 errors will present. 
The F-value tuning coefficients and thresholds that were ultimately chosen are summarized in 
Table 3. 


Table 3: Optimal F-value Tuning Coefficients and Thresholds 


F-value tuning parameters 

F-value Threshold 

Risk Parameters —f 

Quality Parameters —fj 

Tier Level 1-4 

RIC Miss 
Distance 

Pc 

Combined 

Covariance 

Tracking Den- 
sity 

Time Since Last 
Observation 

Tier 

1 

Tier 

2 

Tier 

3 

Tier 

4 

0.0 

1.0 

0.6 

0.1 

0.0 

1.0 

2.5 

2.7 

3.7 


The performance of the values in Table 3 is shown in Figure 9 by the placement of asterisks 
along the sufficient frontier curves (labeled as “solution” asterisks in the figure legend). The de- 
sire was to select a single set of tuning coefficients and four F-value thresholds that would yield 
good performance for prediction of each of the four work tiers. To do this, optimal performance 
for each of the work tier predictions was compromised slightly; this is why the solution asterisks 
do not lie precisely on the sufficient frontier but slightly behind it. 

The actual rate of significant events detected versus false-alarms is summarized in the Table 4 
below. The large number of type- 1 and type-2 errors encountered, even for the selected optimum 
points, is somewhat dispiriting, but it must be remembered that the F-value is trying to predict a 
process that is heavily subjective. 
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Table 4: Performance of the Optimal Set of F-value Tuning Coefficients 



Significant Events Detected 

False - Alarm Rate 

Tier Level 1 

130/151 (~ 7 out of 8) 

1116/4627 ( ~ 1 out of 4 ) 

Tier Level 2 

26/33 (~4 out of 5) 

322/1631 (~ 1 out of 5 ) 

Tier Level 3 

19/24 (~ 4 out of 5) 

203/976 (~ lout of 5 ) 

Tier Level 4 

11/12 (~ 9 out of 10) 

79/633 (-lout of 8) 


ALTERNATIVE CONSTRUCT: PROBABILITY OF COLLISION 

Some analysts maintain that P c alone is an adequate quantifiable metric of satellite conjunction 
risk level, and this is not an unreasonable position, for in principle all the relevant data (miss dis- 
tance, object size, and OD uncertainty as represented by the covariance) are included in the calcu- 
lation. So it is a natural question whether using the P c alone, in the same manner as the F-value, 
can produce a competitive result. The same iterative brute force process as previously discussed 
was used, but the largest calculated P c value throughout the evolution of an event, rather than the 
calculated F-value, was chosen to be the representative datum for each event. No tuning coeffi- 
cients were thus necessary. The P c candidate thresholds chosen for the process ranged from 10 10 
to 0.9 in logarithmically-determined size increments. The optimal set of P c threshold values 
emerging from the analysis is given below in Table 4 and the associated sufficient frontier plot in 
Figure 10. 

Table 5: Optimal P c Threshold Values 


P c Threshold 

Tier 1 

Tier 2 

Tier 3 

Tier 4 

2.0 * 10 -6 

1.0 * 10 -3 

1.5 * 10' 3 

2.0 * lO -3 



Figure 10: Work Tier Predictions Using Only P c Data 
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The actual rate of significant events detected versus false-alarms, using only P c data, is sum- 
marized in the Table 6 below. 

Table 6: Performance Using P c Only Data 



Significant Events Detected 

False - Alarm Rate 

Tier Level 1 

130/151 (~ 7 out of 8) 

929/4627 (~ 1 out of 5) 

Tier Level 2 

20/33 (~ 3 out of 5) 

59/1709 (~ 1 out of 29) 

Tier Level 3 

17/24 ( ~ 7 out of 10 ) 

62/1709 (~ lout of 28 ) 

Tier Level 4 

9/12 (~ 3 out of 4) 

38/1720 ( ~ 1 out of 45 ) 


Here, the performance using only the P c data is very similar to that of the F-value, with some 
minor differences. The P c approach slightly outperforms the F-value in predicting work tier level 
1 (or higher) but underperforms the F-value for prediction of work tiers 2, 3, and 4 (or higher). It 
is difficult to make a definitive choice between the two on these results alone, so one should ex- 
amine the additional consideration of event stability (next section) before making any final rec- 
ommendations. 

F-VALUE TUNING PART III: EVALUATION STABILITY 

Each unique event on average produces several OCM updates, and what is desired is that as 
large a number of these OCMs as possible produce the same correct evaluation (in terms of pre- 
dicted work tier) by the evaluating factor, whether it be F-value or P c As an illustrative example, 
consider an event that produced several OCMs; for each of these OCMs, there is an associated F- 
value and P c that have been calculated. Each work tier level has its respective F-value and P c thre- 
sholds found in Tables 3 and 5; the F-value or P c evaluations that exceed each threshold would 
indicate that the event belongs to that work tier. The stability evaluation process iteratively goes 
through each unique satellite conjunction event for all four different significant work tiers and 
flags a particular OCM update within an event as a success if the calculated F-value or P c 
matched the assigned work tier level for that event; otherwise it is flagged as a mismatch. If all of 
the OCMs for an event produce the correct work tier level, then the stability index for that event 
would be 100%; if only 75% of the OCMs produced a prediction that aligned with the actual 
work tier level, the stability index would be 75%, &c. Of course, percentages closer to 100%, on 
a 0-100 scale, indicate more stable correct evaluations over event history. 

For this type of data result, it is most informative to display the result as a cumulative distribu- 
tion function (CDF) plot. Figure 1 1 shows the stability performance comparison between F-value 
and P c Reading the graph is not entirely straightforward, so a more detailed explanation is pro- 
vided below. The graphs have been reworked somewhat to put them in a more standardized CDF 
form. 

Beginning with the upper left plot, the CDF values that lie on the y-axis give the number of 
events characterized with perfect stability (100% agreement among OCMs for a single event). As 
can be seen, the F-value outperforms the P c slightly at this point (80% to 77%), and it maintains 
this edge at the 50% stability (x-axis) point as well (88% to 84%). For work tier levels 2 and 3 
(upper right and lower left), the situation is inverted in that the P c outperforms the F-value slightly 
at the y-axis (100% stability situation); but by the 50% level both lines have converged, and there 
are essentially no events worse than this level anyway. For work tier level 4, there are essentially 
no non-100%-stable cases for either parameter. 
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While the F-value fares marginally better in some circumstances and the P c in others, both ap- 
proaches render essentially the same performance. Perhaps one could say that the F-value’ s better 
performance vis-a-vis stability (which is in the work tier level 1 case) is more significant because 
the separation between work tier level 0 (a non-significant event) and greater than work tier level 
0 (a significant event) is the most useful distinction to make at the point at which the event is in- 
itially detected (usually seven days from TCA), so a more stable performance is desirable in this 
situation. But even if one grants this, the improved performance cannot be said to be significant. 
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Figure 11: Work Tiers 1-4 Prediction Uniformity 


CONCLUSION: DISCUSSION AND FUTURE WORK 

Conjunction assessment risk analysis is the process of determining if a predicted close ap- 
proach between two orbiting objects poses a risk of collision to those objects. The probability of 
collision computation emerged as a metric for determining the likelihood of a conjunction result- 
ing in a collision. It has been observed operationally at NASA/GSFC that the accuracy of P c as a 
risk metric is subject to necessary understanding of the underlying data used in the computation. 
Conjunction risk analysis, therefore, is this process of qualitatively and quantitatively assessing 
the close approach data. 

One construct employed at NASA/GSFC for quantifying this conjunction risk, including the 
qualitative and quantitative assessment of the close approach data, is the F-value. The F-value, in 
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its present form, is a heuristic model that uses traditional close approach data, such as miss dis- 
tance and P c , but also the required understanding of conjunction metadata, such as tracking data 
density, to more thoroughly quantify conjunction risk. The analysis presented here is an attempt 
to tune that heuristic model with operational analyst data. While the analysis does show success 
in the F-value being able to discriminate between those conjunction events warranting further 
analysis and other non-risky conjunction events, it also shows that the current construct may need 
to be re-visited. 

Ideally, the F-value would be tuned to be able to discriminate between each of the work tiers. 
The inability of the tuning presented here to meet this goal is likely a result of the limited data set 
used to tune the model. Of the 4627 conjunctions used in this analysis, only 151, or less than 5%, 
of these events were in work tier level 1 or higher. Since the majority of the data resided in work 
tier level 0, the tuning may in fact be biased to that data set. This lack of high work tier events is a 
result of using the operational data set, since high work tier events do not make up the majority of 
identified conjunctions. 

Another potential concern with the tuning data available is attempting to gauge the risk asso- 
ciated with the conjunction by the amount of work used to analyze the conjunction via the work 
tier. As stated, the work tiers approach was adopted because no truth measurement of risk is 
available from the operational data set. Flowever, the work tiers represent the amount of work that 
went into a conjunction event and it is assumed that this amount of work correlates to the amount 
of risk. While this correlation is, in general, is true in operations, exceptions do exist. For exam- 
ple, a satellite with no propulsive capability could observe a very high risk conjunction; however, 
since there is no capability to mitigate the conjunction, the work tier would never get past a level 
two, where the P c or the F-value would capture that true, high conjunction risk. The data set was 
not analyzed for the frequency of this type of event. 

While the analysis presented here does not show that the F-value can be tuned to discriminate 
between all work tiers, the F-value as presented in the original paper 3 has demonstrated success in 
predicting and communicating stable conjunction risk in CARA operations at NASA/GSFC. 
Moreover, for many conjunction events, analysis experience continues to show that for the P c 
alone to do an adequate job at summarizing conjunction risk, the need to examine the quality and 
confidence of the solutions used in the close approach prediction process cannot be overlooked. 
Therefore, the F-value concept as a risk metric is still believed to be valid and useful. 

Although this tuning methodology did not provide the ability to discriminate between high 
work tier events, other tuning methodologies are under investigation. First, whether the tuning is 
being biased by the low percentage of high work tier events needs to be addressed. The current 
analysis can be repeated with a smaller data set that evenly distributes the number of events in 
each work tier. If the operational data set is not sufficient, simulated conjunction data may need to 
be produced and used in the tuning effort. Although simulating conjunction data may be more 
difficult, a truth risk metric (e.g., true miss distance) can be determined for each simulated event. 
Tuning the F-value to the simulated true risk, versus the work tier proxy for risk, would be an 
added benefit. The F-value also makes an assumption of linear membership functions which were 
developed to match empirical experience, and constant weighting determined through the tuning 
process. It is suggested that perhaps the linear mapping and weighting are not sufficient. Neural 
networks are frequently used to determine patters from data sets and these tools may outperform 
the current techniques in associating conjunction data to risk. 
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